Controlled experimentation in continuous experimentation: Knowledge and challenges

Continuous experimentation and A/B testing is an established industry practice that has been researched for more than 10 years. Our aim to synthesize the conducted research. We wanted find core constituents of a framework continuous solutions are applied within field. Finally, we were interested in challenges benefits reported experimentation. forward snowballing on known set papers identified total 128 relevant papers. Based this performed two qualitative narrative syntheses thematic synthesis answer research questions. The include processes as well supportive technical organizational infrastructure. found literature synthesized nine themes, e.g. experiment design, automated experiments, or metric specification. Concerning experimentation, analysis cultural, organizational, business, technical, statistical, ethical, domain-specific challenges. Further, study concludes mostly implicit studies. yielded large body knowledge published presented recommended infrastructure process models, guidelines mitigate challenges, what problems various solve.